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ABSTRACT 


This case study analyzes the application of the Cleanroom software development 
methodology to the development of production software at the 
tics and Space Administration/Goddard Space Flight Center (NASA/GSFC). The 
Cleanroom methodology emphasizes human discipline in program verification to 
produce reliable software products that are “right the first time. 

Preliminary analysis of the Cleanroom case study shows that the method can be 
applied successfully in the FDD environment and may increase staff productivity 
and product quality. Compared to typical Software Engineering Laboratory (SEL) 
activities, there is evidence of lower failure rates, a more complete and consistent 
set of inline code documentation, a different distribution of phase effort activity, 
and a different growth profile in terms of lines of code developed. 
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EXECUTIVE SUMMARY 


E.l INTRODUCTION 

The Software Engineering Laboratory (SEL) at the Ac effec- 
Administration/Goddard Space Flight Center (NASA/GSFC) investigates the enec 

tiveness of software engineering technologies when applied I to eve h 

aDDlications software within GSFC’s Flight Dynamics Division (FDD). One such 
technology^ Currently being examined by the SEL is the Cleanroom software deve - 
ooment methodology. There are several significant differences between the SEL 
Cleanroom model tnd the standard SEL development methodology, including the 

following items: 

• Cleanroom testers and developers are on completely separate teams. 

• Cleanroom developers have no access to the mainframe computer for 
compilation and testing purposes. 

• Cleanroom developers rely on code reading instead of unit testing to 
verify correctness of the software prior to system testing. 

• Cleanroom testers use a statistical testing approach. 

E.2 OBJECTIVES AND SCOPE 

This case study analytes the application of the Cleanroom 

a large ground support system used in mission support of attitude determinatio 
reoufremems The system selected for the study was the Coarse/Fine Attitude 
Determination Subsystem (CFADS) of the Upper A»i«ptee Satellite 
fUARS). The completed system contained approximately 3 , 
primarily FORTRAN code. The major goals of the study were to 

. Assess the process used in the SEL Cleanroom model with respect to 
team structure, team activities, and effort distribution 

. Analyze the products of the SEL Cleanroom model and determine the 
impact on measures of interest, including reliability, productivity, overall 
life-cycle costs, and software quality 

• Analyze the residual products in the application of the SEL Cleanroom 
model, such as fault distribution, error characteristics, system growth, 
and computer usage 


E.3 PRELIMINARY ANALYSIS 


The following statements summarize this report’s findings concerning preliminary 
analysis of the Cleanroom methodology and its application in the FDD. 

• The project members were able to successfully apply a tailored version of 
the Cleanroom methodology. 
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• The Cleanroom methodology enhances a “team” development approach 
and minimizes individual programming styles. 

• Preliminary analysis of key measures indicates an increase in productiv- 
ity and reliability and a decrease in rework effort, as compared with 
typical SEL projects. 

• The Cleanroom effort distribution shows a significant increase in design 
effort and decrease in coding effort. 

• Informal design reviews held by the development team appeared tqJse an 

effective method of early fault detection. ^ 

• Less than one-third of the faults uncovered in the code-reading process 
were found by multiple code readers. 

• The breakdown of effort spent in code writing versus code reading was 
approximately even, as compared with a typical SEL ratio of 6 to 1 in 
favor of code writing. 

• All team members indicated a willingness to reapply the methodology on 
future projects. 

E.4 CONCLUSIONS 

Based on the findings indicated in the preliminary analysis of the project, the fol- 
lowing conclusions have been drawn: 

• The Cleanroom methodology can be applied successfully in the FDD en- 
vironment, but additional tailoring is required. 

• The methodology had a favorable impact on all key measures of interest. 

• Early concerns, such as the team’s experience level, the unstable specifi- 
cations environment, and the psychological impact of the methodology on 
team members, appeared to have little or no impact on the application of 
the methodology. 

• The separation of teams forced the developers to apply a more thorough 
effort in design and code verification. 

• The impact of the testing approach cannot be analyzed fully until the 
project has completed its formal acceptance testing phase. 

• Based on the favorable results found here, further studies are called for. 
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SECTION 1-INTRODUCTION 


This case study analyzes the application of the Cleanroom software development 
methodology to the development of production software at the National Aeronau- 
tics and Space Administration/Goddard Space Flight Center (NASA/GSFC). The 
case study involves the methodology’s application on part of a large ground sup- 
port system used in mission support of attitude determination requirements. 

In addition to describing the Cleanroom methodology, this paper analyzes its appli- 
cation in comparison to the existing Software Engineering Laboratory (SEL) meth- 
odology used in GSFC’s Flight Dynamics Division (FDD) (References 1 and 2). 
The analysis covers the phases from project planning through system test. Areas 
of analysis include the tailoring and use of the method, as well as effort, defect, 
and productivity statistics. The emphasis of this study is on understanding the 
methodology and its applicability to the SEL environment, rather than on a de- 
tailed assessment in relation to other development methodologies. 
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SECTION 2— THE CLEANROOM METHODOLOGY: HISTORY, 
DESCRIPTION, AND PRIOR EXPERIENCES 


The Cleanroom software development methodology (References 3, 4, 5, 6, 7, and 
8) was conceived in the early 1980s by Dr. Harlan Mills at IBM. The term Clean- 
room” originates in the integrated circuit (IC) production process, where ICs are 
assembled in dust-free “clean rooms” to prevent the destructive effects of dust. 
When applying the Cleanroom methodology to the software development process, 
the primary focus is on defect prevention rather than defect removal. 

The goal of Cleanroom is to use a structured development process to build a prod- 
uct that is “right the first time,” instead of using the test process to reach a desired 
level of reliability. The essential elements of Cleanroom include an emphasis on 
human discipline and a stress on the use of a structured development approach. 
These elements are enforced in a variety of ways. First, there is a complete sepa- 
ration of development and test activities. Second, there is a reliance on “reading 
for correctness by the developers. Third, the purpose of testing is for quality 
assessment rather than for debugging or finding defects. Finally, a top-down de- 
velopment approach, with the use of stubs, is followed to allow for early assess- 
ment of product quality. Figure 2-1 illustrates the organization needed to follow 
the disciplined approach necessary for Cleanroom. These efforts result in a more 
structured development process, which appears to be the direction of software 
engineering. 

In previous uses of Cleanroom (References 5, 6, 8, and 9), the actual development 
activities were based on the IBM Systems Integration Division (IBM-SID) model for 
software development, which has its foundation in structured programming (Refer- 
ence 10). Using top-down design and development, a system was divided into 
increments, which allowed developers to concentrate on small parts of the system 
at any one time. The formal design process included the use of state machines to 
help modularize the system and to support the concept of information hiding. 
Development progressed through stepwise refinement, and each successive step 
was verified by stepwise abstraction to ensure correctness. In addition, review/ 
inspection activities occurred at various milestones (Reference 11). The purpose 
of the verification and review activities was not to find defects but to confirm 
correctness. Since verification and reviews were perceived as constructive (posi- 
tive) activities, team development was reinforced. 

Testers for Cleanroom projects used a statistical testing approach (References 3, 4, 
11, and 12). As with the development process, the purpose of the test process was 
to confirm correctness and project future reliability, rather than to find defects. 
For this reason, a black-box testing method— specifically, statistical testing— was 
preferred to a white-box testing approach (Reference 4). Data for test cases were 
selected according to the operational profile of the system. All inputs to the 
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igure 2-1. The Cleanroom Development Process 








system were determined, and a probability distribution of possible values for each 
input was calculated. Test cases were then statistically generated. Top-down de- 
velopment allowed system test to begin once the first increment was submitted. In 
addition, since data were statistically generated, it was possible to determine and 
project the reliability of the system in terms of mean time to failure (MTTF). 

As previously stated, this case study was not the first to use the Cleanroom meth- 
odology. Cleanroom has been used for a few projects at D3M-SID and in a con- 
trolled experiment at the University of Maryland. One of the Cleanroom projects 
developed at IBM was the COBOL Structuring Facility (COBOIVSF) (References 7 
and 8). COBOL/SF is a language product consisting of 80,000 executable lines of 
PL/I. The developers were hired directly out of college; thus, Cleanroom was the 
first methodology any of the developers used in a corporate environment. During 
development, the goal was to write simple designs and small procedures, with 
walkthroughs used as a substitute for verification. The emphasis was on proving 
the correctness of design, rather than finding errors. All released versions of 
COBOL/SF were characterized by high quality and productivity. 

At the University of Maryland, students in two graduate-level software engineering 
courses participated in a controlled experiment (Reference 9). Student teams were 
organized so that each team would have comparable experience, and the differ- 
ences in the classroom instruction were negligible. A programming language unfa- 
miliar to all students was used to prevent a bias toward a team that had more 
experience in a particular language, and to control unauthorized execution of pro- 
grams by developers. The teams in one class used the Cleanroom methodology for 
development, while teams in the second class were given the same development 
methodology, but also were given the opportunity to test. The submitted code from 
all teams was tested by the teaching assistant, who executed identical test cases for 
each team. The Cleanroom teams were able to apply the development methodol- 
ogy more successfully. They passed more test cases, fulfilled requirements more 
completely, generated less complex code, and had more inline documentation than 
the non-Cleanroom teams. This experiment indicated that an extra piece of tech- 
nology (developer testing) did not necessarily lead to added success. At the con- 
clusion of the experiment, most of the Cleanroom developers said that they would 
feel comfortable using Cleanroom again, although they missed the satisfaction of 
testing their code. Since the teaching assistant handled all testing responsibilities 
for the Cleanroom teams, the satisfaction level of Cleanroom test teams was not 
part of the project assessment. 
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SECTION 3— THE CLEANROOM CASE STUDY 


3.1 THE SOFTWARE ENGINEERING LABORATORY 

The SEL is sponsored by NASA/GSFC to investigate the effectiveness of software 
engineering technologies when applied to the development of applications software 
(Reference 13). It was organized in 1977 with the following goals: 

1. To understand the software development process in a particular environ- 
ment 

2. To measure the effects of various development techniques, models, and 
tools on this development process 

3. To identify and apply improved methodologies in the GSFC environment 

The principals of the SEL are the Systems Development Branch of the FDD of 
NASA/GSFC, the Computer Sciences Department of the University of Maryland, 
and the Systems Development Operation of Computer Sciences Corporation. Over 
the past 14 years, the SEL has investigated numerous techniques and methods over 
dozens of projects in order to understand and improve the software development 
process in the FDD environment (References 1, 2, 13, 14, 15, 16, and 17). 

The Cleanroom methodology was selected to be studied in the SEL for a variety of 
reasons. The SEL was interested in optimizing the software development effort 
and improving the effectiveness of software testing, thereby reducing the rework 
effort, which encompasses a significant portion of the FDD development effort 
(Reference 18). Cleanroom also displayed the potential to increase software qual- 
ity and reliability without impacting productivity, an area of interest in any soft- 
ware environment. This experiment represented an opportunity to increase the 
SEL’s understanding of Cleanroom in the FDD production environment, rather 
than an academic environment, by a group other than the originators of the meth- 
odology. 

3.2 THE PROJECT AND ITS ENVIRONMENT 

Development was carried out in the standard FDD environment, which is well 
understood based upon a large number of prior studies. The development process 
begins upon delivery of the requirements and specifications document. The re- 
quirements and specifications are generated by an organization separate from the 
development organization, and they adhere to a standard format familiar to both 
groups. The functionally oriented specifications are highly algorithmic and consid- 
ered to be of good quality. However, due to the continually evolving characteris- 
tics of the spacecraft hardware and system architecture, the requirements and 
specifications require frequent modifications. On typical ground support systems, 
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approximately 300 formal inquiries typically are generated by the development 
organization, and 150 to 200 formal specification updates are required over the 
development life cycle. 


The project selected for the Cleanroom case study at GSFC was the Coarse/Fine 
Attitude Determination Subsystem (CFADS) of the Attitude Ground Support Sys- 
tem (AGSS) for the Upper Atmosphere Research Satellite (UARS). The size of 
the CFADS system was initially projected to be about 22,000 FORTRAN source 
lines of code (SLOC), which was approximately 12 percent of the entire AGSS. 
The development environment was an IBM mainframe running the multiple virtual 
storage (MVS) operating system, and the remaining subsystems of the AGSS were 
developed using the standard SEL development methodology. There were numer- 
ous “design drivers,” factors related to the project that limited design options. The 
most significant was the interactive graphics system, which limited the ways some 
data could be bound to procedures. Development for the AGSS began in 
November 1987 and system testing was completed in March 1990. The develop- 
ment of CFADS began in January 1988 and system testing was completed in 
January 1990. System milestones were somewhat different in these projects, as 
illustrated in Figure 3-1. In addition, the CFADS consisted of six builds, with one 
build for each subfunction of approximately 5 thousand source lines of code 
(KSLOC), as opposed to one build for every 60 to 80 KSLOC, as is traditionally 
done on AGSS development in the FDD environment. As seen in Figure 3-1, the 
CFADS did not have a separate acceptance test process, but was integrated into 
the AGSS before acceptance testing began in March 1990. 


The project was staffed by a total of seven different NASA/GSFC personnel. The 
project began with four developers, but dropped to three during design when one 
team member left NASA. The test team was composed of two people, but staffing 
was briefly increased to three when other work commitments prevented the origi- 
nal testers from allocating the planned level of effort. In addition to testing the 
system, the test team also served as library managers. All personnel were also 
working on other projects simultaneously during the CFADS effort. These other 
responsibilities would often take time and attention from the case study. Addition- 
ally, these other projects used methodologies other than Cleanroom, so staff mem- 
bers would often need to use two development methodologies during the same day. 


Figure 3-2 compares the average level of experience for the Cleanroom team 
members with that of a typical SEL project team member. Most of the Cleanroom 
team’s experience was as members of team efforts, with some time also spent 
managing projects. All of their development experience was with the standard 
SEL methodology in the FDD. 
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Figure 3-1. System Schedules for CFADS and AGSS 
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Figure 3-2. Experience Comparisons Between the SEL Cleanroom 
Project and Typical SEL Projects 


5810 


3-4 



3.3 THE GOALS 

The major goals of the SEL experiment were as follows: 

1. Assess the process used in the SEL Cleanroom model with respect to 
team structure, team activities, and effort distribution 

2. Analyze the products of the SEL Cleanroom model and determine the 
impact on measures of interest, including reliability, productivity, overall 
life-cycle costs, and software quality 

3. Analyze the residual products in the application of the SEL Cleanroom 
model, such as fault distribution, error characteristics, system growth, 
and computer usage 

Additionally, other minor goals such as assessing the code-reading activity of the 
SEL Cleanroom model were defined during the life cycle. 

3.4 EARLY CONCERNS 

Four major concerns were expressed by management and development personnel 
very early in the project: 

• The team’s inexperience in the project’s application area 

• The impact of unstable requirements and specifications on the methodol- 
ogy 

• Coordination with the main AGSS, which was developed without Clean- 
room 

• The psychological impact of the methodology on the team members 

Despite the team’s general experience in the FDD environment, this was the first 
time any of the project members had worked in the CFADS application domain. It 
was also the first time any member had used a methodology different than the 
typical SEL methodology. 

As previously stated, the FDD environment is one in which modifications to the 
initial specifications are often necessary. This specification instability was of par- 
ticular interest to the Cleanroom experiment in view of the emphasis on developing 
software “right the first time.” There was also concern for the potential impact on 
rework effort and possible consequences on the timely delivery of builds for the 
test team. Frequently, the delivered specifications are not as detailed as might be 
desired, since assumptions are made based on past FDD projects, and items are 
negotiated between analysts and developers during early design. 

Additionally, coordination between the CFADS and the remaining AGSS needed to 
be carefully planned, noting possible conflicts caused by using different 
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development methodologies. Formal reviews needed special consideration, as did 
common libraries used by both groups. 

Finally, there was concern regarding the psychological impact of the team separa- 
tion and limitation of each team’s activities. The development team and testing 
team had specific tasks and guidelines to follow, and activities for each were care- 
fully defined. There was concern that the developers would attain only minimal 
satisfaction from releasing a product that they would not be able to test or see 
executed. There was a parallel concern that the test team would experience a void 
by testing a product without having participated in the design and development. 

3.5 DATA COLLECTION 

Project data collection methods fell into four categories: forms, automated data 
collection, subjective data collection, and postdevelopment tools. The primary 
source of quantitative data was the set of standard SEL forms used in the FDD 
environment for all SEL projects (Reference 16). These forms address a wide 
variety of issues, from project estimation to personnel resources and change de- 
scriptions. In addition to the standard forms, a few new forms were designed to 
gather additional data and fulfill functions unique to this project. Information such 
as source code growth (system SLOC), source changes (module version updates), 
and computer usage (central processing unit, or CPU, hours) are automatically 
provided by several tools running in the host machine. To gather qualitative data, 
interviews and informal discussions were held. An observer from the University of 
Maryland familiar with the IBM-SID Cleanroom model was present for 10 to 
15 hours a week at GSFC. The observer’s task was to resolve questions pertaining 
to Cleanroom, tailor the methodology for the environment, ensure that the method- 
ology was being used properly, and gather data and give real-time feedback to the 
developers and testers. At the conclusion of the project, standard tools were run to 
gather system statistics, including detailed component attributes and source code 
characteristics such as size and complexity. 
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SECTION 4— A TAILORED SEL CLEANROOM MODEL 


The Cleanroom methodology had to be tailored to the FDD development environ- 
ment, which is based on the waterfall model (References 1, 2, and 15). The 
tailored methodology needed to preserve the salient features of Cleanroom, but 
also needed to be easily adapted to the FDD environment. Some features of IBM- 
SID Cleanroom, which the developers and testers were not able to use because of a 
lack of experience in IBM-SID methods, needed to be reevaluated. Other charac- 
teristic SEL activities required modification to simulate IBM-SID Cleanroom fea- 
tures. Over a period of time, a version of Cleanroom was defined that seemed to 
best fit this environment. This period of time extended into development, as the 
initial Cleanroom description evolved to account for new problems encountered in 
the environment. The tailored methodology was referred to as the SEL Cleanroom 
model. 

There were several significant differences between the SEL Cleanroom model and 
the standard SEL development methodology: 

• Cleanroom testers and developers are on completely separate teams 

• Cleanroom developers have no access to the mainframe computer for 
compilation and testing 

• Cleanroom developers rely on code reading instead of unit testing to 
verify correctness of the software prior to system testing 

• Cleanroom testers use a statistical testing approach 

Table 4-1 highlights differences between the SEL Cleanroom model and the stand- 
ard SEL development methodology. The SEL Cleanroom model used the standard 
SEL guidelines (References 1 and 2) for top-down design and development, but 
had more increments than a similar project using the standard methodology would 
have had. The project’s requirements and specifications analysis process was 
strongly emphasized. The design process was a combination of SEL and IBM-SID 
activities, with attempts to use state machines, detailed program design language 
(PDL), and a generic design. High level designs were reviewed at various mile- 
stones. Team design reviews confirmed correctness during low level design, while 
redundant sequential code reading did the same during the coding phase. Sequen- 
tial code reading differs from code reading by stepwise abstraction in that lines of 
code are read in physical order, not by functional hierarchy. The developers used 
sequential reading because it is commonly used in the FDD environment, and the 
training schedule did not offer sufficient time for skill development in reading by 
stepwise refinement. 

All development was done at desks and on personal computers (PCs), with files 
transferred from the developers to the testers via floppy disks. The testers 
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Table 4-1. Process Comparisons Between the SEL Cleanroom and 
Standard SEL Development Models 



Organization 

Design 

Quality 

Control 

Code 

Quality 

Control 

Testing 

Strategy 

SEL 

Cleanroom 

Separate 
development 
and test 
teams 

Team 

design 

reviews 

Sequential 
redundant 
code read- 
ing for 
verification 

Statistical 

testing 

Standard 

SEL 

Single de- 
velopment 
and test 
team 

PDL 

reading 

Code read- 
ing and unit 
testing 

Integration 
and system 
testing 
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managed the libraries and uploaded code from the PC to the mainframe in order to 
build and test the system. This process ensured that the developers did not com- 
pile or unit-test their code, as they were not given any access to the mainframe. A 
statistical testing approach was to be used to validate the code, and failures were 
reported back to the development team. The developers then identified the error 
source and took the appropriate corrective action. Table 4-2 compares develop- 
ment and test team responsibilities in the SEL Cleanroom model. 

4.1 TRAINING AND PREPARATION 

Before project development began, both teams attended a 1-week tutorial on the 
Cleanroom methodology. The training was also attended by the project’s supervi- 
sors. Lectures were given by Dr. Victor Basili of the University of Maryland, and 
Mr. Michael Dyer and Mr. F. Terry Baker, both of IBM-SID. Sessions discussed 
Cleanroom in general, and emphasized the IBM-SID method of software develop- 
ment. The classes consisted of lectures followed by question-and-answer sessions. 
Later, there were follow-up sessions by Mr. Dyer on statistical testing for the test 
team, and a presentation by Dr. Basili on verification by stepwise abstraction for 
the developers. 

The purpose of the training sessions was twofold. First, it allowed the CFADS 
team the opportunity to learn as much as possible about the theory behind the 
methodology and how Cleanroom has been done in the past. Second, it served to 

reduce some of the misconceptions and apprehension of the team members. The 
cost of the training, in terms of CFADS team effort, was 4 percent of the total 
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Table 4-2. Comparison of SEL Cleanroom Team Responsibilities 


Schedule 

Project 

Phase 

Development Team 

Test Team 

1/88 

Training 

Attend 1-week tutorial 
Attend additional sessions on 
design and code reading 

Attend 1-week tutorial 
Attend additional sessions on 
test case generation 

2/88- 

11/88 

Design 

Analyze requirements and 
specifications, submitting 
questions as needed 
Create high level design using 
abstract state machines 
Create low level design using 
PDL 

Review designs 

None 

- 

Pretest 

None 

Analyze requirements and 
specifications, submitting 
questions as needed 
Analyze specifications to 
understand functionality of 
the system 

Determine test items and 
passage criteria 
Determine system inputs and 
distributions 

12/88- 

12/89 

Code 

Write code components 
Read code independently 
Submit code to testers 

None 


Test 

Respond to failures encountered 
by testers 

Correct and reverify code 

Generate test cases 
Handle all configuration 
control activities 
Compile components 
Link components 
Execute test cases 
Validate results 
Return code to developers 
for correction 
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hours expended on the project. Overall, the team members were happy with the 
tutorial, but felt that the addition of a laboratory exercise would have made the 
activity more effective. At the conclusion of the training sessions, two project 
members felt skeptical about proceeding with Cleanroom, two were cautiously con- 
fident concerning the methodology’s application, and two members seemed confi- 
dent that the methodology could be applied successfully in the FDD environment. 
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SECTION 5— APPLICATION OF THE CLEANROOM 

METHODOLOGY 


5.1 PREDESIGN 

The predesign phase, accounting for 30 percent of the total design effort. was 
given greater emphasis than on a typical FDD project. The major activity in pre- 
design was requirements analysis, where the requirements and specifications were 
studied in order to understand the problem domain, resolve ambiguities, and make 
corrections. The FDD has a history of reusing requirements and specifications 
from previous projects, and these documents frequently assume an understanding 
of the systems developed for previous satellites as a base for describing the present 
project. Additionally, the documentation is often dynamic, and changes can occur 
during any phase of development. Since Cleanroom strives to develop software 
that is “right the first time,” the requirements and specifications had to be more 
complete, so the requirements analysis activity needed to be more thorough. Team 
meetings were held to ensure that all development and test team personnel had a 
common understanding of the documents. Where ambiguities or errors were 
found, clarifications were requested. The result was an improved set of documen- 
tation for the developers and testers to work with. 

5.2. HIGH LEVEL DESIGN 

The remaining 70 percent of the design effort was divided into design creation 
(49 percent) and design review (21 percent). The high level design process at- 
tempted to emulate the high level design activities used at IBM-SID, encapsulating 
data and functions as state machines. Use of state machine concepts such as 
information hiding and data abstraction helped the developers in the design proc- 
ess, although there are few explicit signs of these concepts in the design and code 
of actual modules. One reason for the inability to actually implement state ma- 
chines may have been the functional specifications, which strongly implied a func- 
tion-oriented design. In addition, the developers’ lack of experience in using state 
machines as a design representation may explain why they were able to use state 
machines conceptually but not concretely. 

Because of problems in the early stage of development, such as the inability to 
build concrete state machines, some members of the development team again ex- 
pressed concerns with the methodology. This may have been attributable to the 
fact that the methodology was still being tailored and was not in a clear, final 
form. This proved to be frustrating for some of the developers, as they could not 
see a complete life-cycle model and found it difficult to work on one step of the 
methodology when the next step potentially could be redefined. Many of the atti- 
tudes changed as the process became more complete and familiarity increased. 
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Tangible results also helped increase team confidence in the Cleanroom methodol- 
ogy. 

Since the process was being tailored as the project progressed, it seemed logical to 
follow the IBM-SID Cleanroom model. Although the IBM-SID and SEL environ- 
ments are similar in their use of disciplined life-cycle development activities, the 
level of formality employed during the activities is different and prevents a blanket 
use of the EBM-SID model. For example, the EBM-SID model bases its design 
review activities on the formal Fagan inspections (Reference 11), while the SEL 
Cleanroom model relied on informal peer review techniques familiar to the envi- 
ronment to verify design correctness. 

The content and schedule of builds were also defined during this phase. The 
project leader, a member of the development team, worked together with the test 
team to determine the number of requisite builds and the functions that would be 
contained in each. This joint effort allowed the builds to be viewed from two 
perspectives and attempted to keep both teams sufficiently active during the entire 
implementation and testing phases. An effort was made to schedule builds so that 
the development team would complete a current build at approximately the same 
time as the test team finished with the previous one. 

5.3 LOW LEVEL DESIGN 

Since developers were not permitted to test their code, the developers inferred that 
the design had to be of high quality. This encouraged them to write more detailed 
PDL, and to make a strong effort to reduce their individual programming styles, 
opting for a common design to facilitate the design review process. Because devel- 
opers used the state machine concept, the system was modularized differently than 
it normally might have been, although actual state machines were never imple- 
mented. 

The team design review process was simple and successful. A few days before a 
scheduled review, the designs for a set of related modules were distributed to the 
other developers. The design for each module consisted of a standardized prolog, 
calling sequence information, and the PDL. The appropriate baseline diagrams 
also were distributed, along with any related COMMON blocks and general notes. 
The designs were studied individually by the other developers, which encompassed 
15 percent of the design effort, and faults were noted. During the design reviews, 
which accounted for 6 percent of the design effort, faults were discussed and cor- 
rections suggested. A set of modules would remain in the design review process 
until the development team determined that there was no need to review the design 
again. All designs required at least two reviews. 

As a result, the assessment of a representative sample of the CFADS design 
showed an average of 18.4 faults/KSLOC found during the design reviews. This is 
a lower bound, as the designer corrected other faults before and after reviews. 
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Since the SEL does not normally track design faults, it is impossible to evaluate 
this parameter in relation to the existing SEL methodology. The distribution of the 
faults by types along with raw count and percentage of faults found in each activity 
appears in column A of Table 5-1. Qualitatively, the developers were confident in 
the completeness and accuracy of the final designs, and felt they knew and under- 
stood the entire system better than they normally would have on other projects. 


Table 5-1. Fault Distribution by Quality Control Activity 



Activity 


Fault Type 

A 

CD 

C 

D 

Totals 


Design 

Reviews 

Code 

Reading 

Compilation 

Testing 


FORTRAN Syntax 

0.0% 

4.0% 

100.0% 

0.0% 

7.0% 

Control Flow 

20.0% 

8.0% 

0.0% 

12.0% 

12.0% 

Interface 

24.0% 

17.0% 

0.0% 

34.0% 

20.0% 

Data Initialization 

1 .0% 

5.0% 

0.0% 

12.0% 

4.0% 

Data Declaration 

45.0% 

19.0% 

0.0% 

5.0% 

25.0% 

Data Use 

0.0% 

32.0% 

0.0% 

23.0% 

19.0% 

Computation 

10.0% 

9.0% 

0.0% 

1 1 .0% 

9.0% 

Displays 

0.0% 

6.0% 

0.0% 

3.0% 

4.0% 

Total Number of Faults 

542.0’ 

883.0 

74.0 

175.0 

1 ,674.0 

Percent of Total Faults Found 

32.0% 

53.0% 

4.5% 

10.5% 
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5.4 CODING 

The more thorough design process allowed the developers to concentrate solely on 
coding the system during the coding stage, without major impacts due to an incom- 
plete design. Through the first few builds, slightly more than half the effort in the 
coding phase was spent writing code, with the rest in code reading. More time was 
shifted to reading code as failures were found by the testers and corrected by the 
developers, who were no longer writing as much new code. The final relationship 
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was that 48 percent of the total coding effort was expended in code writing, and 
52 percent was spent in code reading. 

The code review process was similar to the design review process. Strict coding 
guidelines were established and adhered to by the developers, although opinions 
regarding the content and flexibility of the guidelines varied among the develop- 
ment team. This standardization aided the code reading process by forcing all 
code to appear similar in style, and made the process of mapping the code to the 
PDL easier. Related modules were coded and listings given to the readers along 
with necessary references. The readers read the code independently in a sequen- 
tial manner, marked faults, and suggested corrections. Code was returned to the 
developer, corrected, and given back to the readers to be reread. Code was ready 
to be delivered to the testers only when no faults were found by the readers. Most 
modules required two or three iterations of the code reading process. 

Redundant sequential code reading resulted in significant numbers of faults being 
found and corrected before the code was actually tested. The readers found and 
corrected an average of 30 executable faults/KSLOC while code reading. An addi- 
tional 10.4 nonexecutable faults/KSLOC were found and corrected. Nonex- 
ecutable faults are faults found in the commentary for the procedure, which could 
not have been found by the testers. These corrections helped to make prologs 
more consistent and complete in relation to the code, and should make mainte- 
nance efforts easier. As with the design reviews, these numbers are lower bounds, 
as each developer found and corrected additional faults before and after his code 
was read by the readers. Surprisingly, of the 30 executable faults/KSLOC found 
by the readers, only 28.5 percent were found by both readers, with no consistent 
pattern between readers in terms of relative effectiveness. This leads to the con- 
clusion that for this project two readers were more effective than one would have 
been, and that sequential code reading allowed readers to read the code in differ- 
ent ways. 

One of the early concerns for the project was the impact of specification changes, 
a common occurrence in the FDD environment. However, this concern was never 
any more of a factor than is typically found in other SEL projects. Subjectively, 
the developers felt making changes of any type was actually easier in the Clean- 
room environment due to the detail and accuracy of the PDL. All software 
changes were reverified, and appropriate PDL updates were made and checked. 

It should also be noted that all reused modules for this experiment were reused in 
their entirety. Any reusable module that required changes was rewritten to follow 
the design and coding standards and passed through the review process as a new 
routine. 

The actual distribution of faults by type, along with the total number of faults and 
percentage per activity, is shown in Table 5-1, column B. Qualitatively, the 
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developers had high confidence in their code. These results also confirm the effec- 
tiveness of reading, as found elsewhere in the literature (References 3 and 9). 

5.5 PRETEST 

As was shown in Figure 3-1, the testing phase ran concurrently with the develop- 
ment phase. The testing process began with a requirements analysis. This activity 
was performed with the developers, to enable the specifications to be reviewed 
from two different perspectives and to promote a common understanding of the 
specifications by both groups. Following the requirements analysis, interaction 
between the testers and developers ended, and did not resume until the actual 
testing activities began. In addition to studying the requirements, pretest activities 
included identifying potential test items, determining system usage profiles, pre- 
paring test data, generating test cases, mapping test items to the test cases, and 
generating expected results for each test case. The pretest phase involved 32 per- 
cent of the total testing effort. 

Test items defined the lowest level of functionality to be evaluated during the test- 
ing process. A test item consisted of one or more functions that mapped directly 
to the specifications. Identification of the specific test items was a subjective proc- 
ess that relied on the testers’ impressions concerning the level of functionality that 
could be verified. Each test item was then mapped to the build where it would be 
available for validation. As previously stated, the test team was directly involved 
in determining the content of each build, so an attempt was made to reasonably 
spread test items throughout all builds. 

These activities were simplified by the functional orientation of the specifications. 
Alterations to the specifications, however, which continued throughout the testing 
activity, resulted in changes to the test items identified. A great deal of care was 
needed to ensure a mapping of the test items to the specifications throughout the 
testing effort. While no interaction was permitted between the testers and develop- 
ers during pretest activities, it was later determined that communication with the 
developers may have been useful in defining the type and level of diagnostic output 
desired to support the testing activities. 

The manner in which the system was expected to be used by an operator (the 
usage profile) was determined by direct observation of operators executing similar 
systems and by previous experiences of the testers. This information, combined 
with specification outlines on user interaction with the system, was used in an 
attempt to develop a statistical profile of system inputs and options. These distri- 
butions were then used as the basis for generating test cases to closely simulate 
actual operator use of the system. While test cases were generated based on usage 
profiles, it was later determined that the variation from test case to test case was 
minimal. This was due to the limited number of inputs required by the system 
(few option selections), the types of inputs required (strong dependencies on data 
and results from outside interfaces), and the low level of user interaction required 
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by the system (the sequential operation of the system). Because of the limited 
variation in the test cases, many were later abandoned during the testing activities, 
as they would have given little additional insight about the overall system quality. 

Once test cases were generated for the system, they were mapped to the subset of 
test items associated with the build being evaluated. Based on the total test item 
coverage by all test cases, the testers were free to generate test cases independent 
of the system usage profile. These test cases isolated critical test items not cov- 
ered in the other test cases, or test items that may rarely be exercised but are 
nonetheless critical. 

The final pretest activities included preparation of test data for the test cases and 
generation of expected results for the test items in each test case. Because the 
Cleanroom build schedule required data earlier than in traditional development 
efforts, the unavailability of the necessary data forced these activities to be post- 
poned until the test cases were executed. 

5.6 TEST 

The remaining 68 percent of the testing effort comprised three distinct activities: 
configuration control, system integration, and test case execution/verification. 

Traditional SEL configuration control practices were followed by maintaining a 
controlled library of all source code for the system on the host machine. However, 
because developers were not permitted to access the mainframe, additional con- 
figuration control responsibilities included the maintenance of a separate con- 
trolled library on a PC. The mainframe control library was used to integrate the 
system, while the PC control library was used to distribute components back to the 
developers, upon their request, for modification. After being placed into the PC 
control library, new or modified components were uploaded to the mainframe con- 
trol library. The extra effort required to manage two libraries was minimal but did 
require careful procedures to ensure consistency between the two. The test team 
was also required to maintain a library of stubs on the mainframe for use in the 
system integration. 

The system integration activity was viewed as a two-step process. The first step 
involved compiling all new and modified components as they were received and 
uploaded by the testers. The results of this compilation were the first opportunity 
to assess the effectiveness of the development method. Of the 101 FORTRAN 
subroutines, which averaged 258 SLOC, 62 percent compiled on their first at- 
tempt. Among those that failed, all but one compiled on the second try. Although 
all faults were obviously due to incorrect FORTRAN usage, over 90 percent were 
directly attributable to typographical mistakes such as missing commas or mis- 
spelled variable names. As might be expected, BLOCK DATA routines displayed 
greater success. Ninety-one percent compiled on the initial attempt, and all that 
failed the first time compiled on the second attempt. Overall, 70 percent of all 
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compilable units compiled successfully on the first try. This result compares favor- 
ably to the 63 percent figure of first compilation success for the first increment of 
COBOL/SF at IBM-SID. The second step of the system integration process was the 
rebuilding of the system to support test case execution. The need to completely 
rebuild the system from scratch each time, rather than linking in only the modified 
components, became evident early in the testing effort. This was due to a depend- 
ency on other development groups, separate from the CFADS effort, for common 
portions of the system. Communication was often inadequate and occasionally 
resulted in outdated versions of the common components being included in the 
CFADS integration process. To promote rapid turnaround when identifying, isolat- 
ing, and correcting faults in the system, the load module was rebuilt frequently, 
often two or more times per week. There was also a greater dependence on system 
stubs, since the entire system was to be integrated and executed much earlier in 
the development process than would be required during a traditional development 
effort. This required the test team to maintain the stubs library on the host and 
include it in the system integration process. 

Test case execution and verification accounted for the majority of the test team s 
efforts. Activities consisted of preparing test data, generating expected results, 
executing test cases, analyzing and reporting results, and supporting the develop- 
ment team’s fault isolation efforts. As stated earlier, test data preparation and 
expected results generation were postponed until test activities began. For the 
generation of simulated test data, the test team relied heavily on an external sys- 
tem developed solely for that purpose. Because data were needed much earlier in 
the Cleanroom effort than in traditional SEL efforts, there were conflicts with the 
development schedule of the data simulation system. This resulted in requests for 
data which were frequently only partially satisfied or exhibited analytic inconsis- 
tencies throughout testing activities. The impact to the testers was most strongly 
felt in the difficulties caused in generating expected results prior to test case exe- 
cution, and the inability to exercise all test items associated with a test case. 

The test team executed the system based on a test case setup. During execution, a 
log was kept that identified the run, documented alterations to the test case, and 
described abnormalities observed. All outputs generated by the system, including 
copies of the interactive screens displayed, were collected and attached to the log 
for future analysis. Alterations to the test cases were required when there were 
deficiencies in the test data or in the functionality of the system, or when modifica- 
tions to the specifications affected the mapping of test items. Functional deficien- 
cies were primarily the result of software that was designed to provide a certain 
level of functionality but failed to do so because of source code errors or limita- 
tions with system interfaces. The limitations often were linked to conflicts with the 
schedule of the interfacing group responsible for developing those components. To 
expedite the testing process, the test team adjusted cases appropriately to test 
around known deficiencies. As these problems were addressed, the required 
functionality was added within the system. The net result for the testers was an 
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inability to execute test cases as specified and to exercise various test items associ- 
ated with a test case. The final solution was to abandon many of the test cases in 
the interest of the project’s schedule and resources (possibly justified, since there 
was little variation from test case to test case), and to postpone Until the last build 
of the system the verification of many of the test items. In several instances, it was 
not until then that the required data became available, the required functionality 
was provided, or the modifications to the specifications were completely under- 
stood to allow for accurate assessment of the code. 

Once a test case had been executed, the results were analyzed and failures re- 
ported to the developers. Analysis of the results required the generation of ex- 
pected results for each test item and a comparison with the computed results. 
Generation of expected results was usually quite tedious and time consuming. The 
comparison of expected results with computed results occasionally resulted in re- 
quests from the testers to modify the diagnostic output in the system. A log kept 
during analysis activities documented any concerns, questions, and problems iden- 
tified and provided a compilation of which test items passed, failed, or could not 
be verified. In addition, a test item status sheet was maintained to track the status 
of each test item. All failures were documented using a Software Failure Report 
(SFR) form, which identified the test case run and described the observed failure. 
To expedite the process, developers were often informed of a run’s results infor- 
mally and potential faults were investigated prior to the generation of the SFRs. 

When a potential failure was identified, the developers attempted to isolate the 
problem and define a solution, with assistance from the testers as required. On 
occasion, developers requested additional runs of the system to assist in their fault 
isolation efforts, and on a single occasion it became necessary to build a test ver- 
sion of the system which was modified directly to isolate the cause of a failure. 
Several problems were also traced to the external interfacing groups. 

The testing process uncovered 3.3 errors/KSLOC. Errors represent the number of 
software changes attributed to error corrections on data collection forms. Errors 
are tracked only for changes that occur after code is placed under configuration 
control. These errors resulted in a total of 175 faults, whose distribution by type 
are listed in column D of Table 5-1. Most faults were identified and corrected the 
day the failure was reported, although some took several days to correct, and a few 
took weeks to correct. Once a fault was isolated, the first correction implemented 
typically resulted in the removal of the failure. As mentioned earlier, while a fault 
remained in the system, an attempt was made to test around the problem. 

After correction or modification was made to the system, test cases or variations of 
test cases were executed to verify the modification. In the case of a failure correc- 
tion, the test case that caused the failure was reexecuted. For other modifications 
due to specification changes, test cases were usually created by the testers to exer- 
cise the specific test items affected. Testing activities continued until all test items 
were verified. 
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SECTION 6— PRELIMINARY ANALYSIS 


The fundamental finding was that the Cleanroom methodology could be used suc- 
cessfully in the FDD environment, as the CFADS team demonstrated. 

The completed system contained approximately 31 KSLOC divided into 
101 FORTRAN subroutines, 33 BLOCK DATAs, 33 COMMON blocks, and 
2 NAMELISTs. Combined with the 2.9 KSLOC for the graphic displays, which 
were not developed using Cleanroom, the entire CFADS totaled 34 KSLOC. Fig- 
ure 6-1 compares the growth history of the CFADS with similar systems developed 
using the traditional SEL approach. As a result of the increased design effort 
code appeared later and system growth progressed much more quickly. Because of 
the incremental development and code reading processes, which forced related 
modules to be read by the developers and submitted to the testers together, the 
SEL Cleanroom growth in Figure 6-1 appeared as a step function. 

The personnel effort distribution during the development process for the Clean- 
room project is somewhat different than the effort expended on other FDD proj- 
ects. As Figure 6-2 shows, the Cleanroom effort spent more time in Design and 
Other (management, meetings, etc.) activities and less time in the Coding area. It 
should be noted again that almost one-third of the Testing effort was spent on 
pretest activities, which resulted in only 18 percent of the entire project’s effort 
being spent actually executing test cases and finding and fixing defects. 


Productivity for the project is approximately 4.9 SLOC per staff hour, from 
predesign through system test. This number compares favorably with the average 
of 2.9 SLOC per staff hour typically found in this environment. 

During the testing phase, the error rate of the system was 3.3 errors/KSLOC, com- 
pared to the 6 errors/KSLOC typically found in the FDD environment. In actual- 
ity, the Cleanroom error figure is artificially high for comparison purposes, as 
some errors that would typically have been found during unit test when using an- 
other methodology are not encountered until the system test process when using 
Cleanroom. Typically, the SEL does not formally track errors found during unit 

testing. 

In analyzing the number of faults found during the various activities of the life 
cycle, 85 percent were found and corrected before any code came under configura- 
tion control. Table 5-1 showed the fault type breakdown by activity. Additionally, 
nearly 90 percent of all faults were removed before the first test case was exe- 
cuted. The majority of the faults found throughout development and test were data 
declaration problems, followed in frequency by data use and interface issues. 
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Figure 6-1. Growth of System in Calendar Time Through System Testing 









The early concerns about using Cleanroom on the project and in the FDD environ- 
ment did not limit Cleanroom’s applicability. 

1. The team’s inexperience in the application area was countered by a more 
complete Requirements Analysis process. The increased early emphasis enhanced 
the development and test teams’ familiarization with the project domain. 

2. The impact of an unstable requirements and specifications environment 
was lessened by a concentrated effort in the team review processes, which encour- 
aged detailed PDL and accurate inline code documentation, and made later stages 
of the Cleanroom methodology less prone to the effect of requirements and specifi- 
cations changes. 

3. Coordination with the main AGSS, which was developed without Clean- 
room, did present some minor problems. Since some software was common to 
both groups, the Cleanroom team was directly affected when modifications were 
made or errors were discovered. Furthermore, because the Cleanroom team ac- 
cessed the common libraries earlier in the development phases, they were often the 
first group to uncover an error, and thereby felt the greatest impact. Additionally, 
communication between the groups was not as frequent or thorough as necessary 
during the initial phases of the projects. Formal mechanisms to aid the communi- 
cation process were added and proved helpful. The Cleanroom team also needed 
to make some adjustments in the format of their design materials in order to par- 
ticipate in the combined formal design reviews. 

4. The application of methodology and team structure seemed to have mini- 
mal impact on the project personnel’s level of professional satisfaction. While 
team members did note that there was an adjustment in the importance of activi- 
ties, they also indicated that the overall success of the project was still of primary 
interest. Overall, the psychological impact of the methodology was not a factor in 
the project’s degree of success. 

Preliminary analysis of the first two goals of the experiment, understanding the 
Cleanroom process and product, has been performed. Additional analysis with 
respect to these goals remains. The results of this analysis will be used to refine 
the SEL Cleanroom model for use on future projects. This packaging will attempt 
to incorporate the successes in this project, while learning from failures or mis- 
takes made in the planning or execution of the methodology’s application. Finally, 
the refinement must be based upon the concepts that can be applied effectively to 
the SEL Cleanroom model. For example, previous studies indicate that reading by 
stepwise abstraction is more consistent with the methodology and may prove, with 
proper skill development, more effective. A redesign of the entire system may 
permit a broader application of state machines. The result of this process will be 
an appropriate SEL Cleanroom model that is more effective in the environment. 
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SECTION 7— CONCLUSION 


Analysis of the Cleanroom case study shows that the methodology can be applied 
in this environment. There are indications of an increase in developer productivity 
and product quality. Compared to typical SEL activities, Cleanroom produced a 
lower failure rate (3.3 errors/KSLOC versus 6 errors/KSLOC), and there are indi- 
cations of a more complete and consistent set of inline code documentation. In 
terms of effort, there is a different distribution of phase effort activity (more time 
in Design and meetings, and less time in Coding) and a different growth profile in 
terms of lines of code developed (code appears much later). 

The Cleanroom case study generated important lessons for application on future 
projects. It is clear that the development effort is a true team effort, and all team 
members must be able to function as such. Additionally, there is visible benefit in 
the adoption of strict design and coding standards for all development team mem- 
bers to follow. These standards should be defined before a project begins, reduc- 
ing debate among team members on personal preferences. It appears that the 
actual standards are not as significant as the consistent application of the standards 
by all participants. It is also recognized that the test team and development team, 
while performing distinct functions, need frequent communication. Both groups 
need to be aware of specification changes, and the developers require input from 
the testers regarding desired diagnostic data for test item validation. Configuration 
control and the transfer of modules between teams also requires a well-coordinated 

effort. 

Subjectively, project personnel felt that the activities associated with the methodol- 
ogy are quite similar to activities typically done in the FDD environment. The 
difference in using Cleanroom is evident in the goals and discipline level associ- 
ated with the activities. All members of both teams indicated that they would be 
willing to use the methodology on future projects. 

There are also a few specific issues that will affect the refinement of the methodol- 
ogy in the FDD environment. Since developers do not have access to the main- 
frame, they must rely heavily on their personal knowledge of the development 
environment. Situations may arise where more time is expended tracking down a 
solution through documentation or a colleague’s recommendation than would have 
been spent in a trial-and-error session on the mainframe. It is also unclear whether 
the design phase should be completed for all builds before any coding begins (to 
isolate design inconsistencies), or if the design and code for each build should be 
completed before design on the next build commences (to produce testable soft- 
ware earlier). 

More analysis of the Cleanroom project is planned, as well as applications on 
future systems. Upon completion of acceptance testing, the appropriate final 
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adjustments to the data on effort, productivity, and quality will be made. Compari- 
sons between the Cleanroom process and products and the process and products 
typically used in the FDD will continue. All the dhta gathered will be further 
analyzed to reach a better understanding of the causes of the results and the con- 
texts in which the data should be interpreted. An appropriately tailored Clean- 
room model for the FDD environment will evolve using the results of this 
experience as a basis. 
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GLOSSARY 


AGSS 

CFADS 

COBOL/SF 

CPU 

FDD 

GSFC 

JBM-SID 

IC 

KSLOC 

MTTF 

MVS 

NASA 

PC 

PDL 

SEL 

SFR 

SLOC 

UARS 


Attitude Ground Support System 

Coarse/Fine Attitude Determination Subsystem 

COBOL Structuring Facility 

central processing unit 

Flight Dynamics Division 

Goddard Space Flight Center 

IBM Systems Integration Division 

integrated circuit 

thousand source lines of code 

mean time to failure 

multiple virtual storage 

National Aeronautics and Space Administration 

personal computer 

program design language 

Software Engineering Laboratory 

Software Failure Report 

source lines of code 

Upper Atmosphere Research Satellite 
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